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Abstract 


Although the field of machine translation has witnessed huge improvements in recent 
years, its potentials have not been fully exploited in other interdisciplinary areas such as 
foreign language teaching. The aim of this paper, therefore, is to report an experiment in 
which this technology was employed to teach a foreign language to a group of students. 
This mixed-method study explores the effect of teaching editing techniques in machine 
translation to a group of Persian EFL university students in an academic writing course. 
Twenty students took part in a 4-day workshop in which one session was devoted to 
teaching editing techniques and three remaining sessions to the use of editing techniques, 
namely, correcting mistakes, removing ambiguities, simplifying structures and combining 
structures. Each session consisted of a pre-test, a training and a post-test. In addition, in 
each session, one key writing point, namely, determiners, paraphrasing and collocations 
were discussed. A questionnaire for candidates’ demographic information and another for 
learning experiences were administered. The results indicated a statistically significant 
improvement in the overall gain score. Further analysis showed a significant improvement 
in the use of determiners in contrast to paraphrasing and collocations. Lack of 
improvement in data driven learning in paraphrasing and collocation seemed to stem from 
weakness in vocabulary and grammatical knowledge in both the mother tongue and the 
target language. Analysis of questionnaire data revealed that the instruction proved to be 
beneficial since it could be easily implemented in correction and confirmation. On the 
whole, it can be concluded that providing the correct type of guidance and feedback on 
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how to use machine translation will indeed have a profound effect on foreign language 
writing skill. 
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1. Introduction 


Online tools such as search engines, dictionaries and encyclopedias have greatly affected 
learning in general and language learning in particular. In foreign language writing 
classes, students are engaged with these online tools and receive online feedback from 
peers and instructors (Kern, 2006; Egbert, 2005). Students write in online collaborative 
environments and teachers check student writings and provide immediate corrective 
feedback. In addition, other online resources such as dictionaries and search engines have 
been implemented by the students since they are easy to use and reflect real life language 
usage (Shei, 2008; Kilgarriff & Grefenstette, 2003). Corpora such as the British National 
Corpus (BNC Consortium, 2007) and the Corpus of Contemporary American English 
(Davies, 2008) have also been integrated into academic writing classes. 


These digital online resources have been successfully integrated into foreign language 
writing classes making data driven learning (DDL) a reality. The potential benefit of DDL 
has been described by Boulton (2010) as encouraging consciousness- raising and noticing, 
leading to better language learning skills and greater learner autonomy in the long run. 
However, since DDL can be both cost-ineffective and intensive, some concerns have been 
expressed. Learners, for example, have to spend lots of time and effort to identify correct 
linguistic form while employing DDL. In addition, many learners may not have access to 
digital devices due to various reasons (Boulton, 2010). 


Many foreign language writing classes in developing countries are still conducted in 
traditional form. The teachers are the suppliers of correct linguistic forms and their 
working load is daunting. In this case, they have to provide general training and feedback 
ignoring individual differences and needs in the writing process. In addition, these analog 
teachers deprive their students of the enormous digital opportunities available to them. 
If these online tools are correctly incorporated into writing classes, in addition to reducing 
the teachers’ workload, they will actively engage students with their writing assignments, 
leading to greater self-confidence in writing (Milton, 2006; Chapelle, 2003). Therefore, at 
this stage we could ask ourselves what digital technologies can be incorporated into our 
writing classes and how? 


In order to identify such technologies, the present study has been devoted to the 
implementation of teaching editing the input for machine translation (ETMT) which to the 
best of the researchers’ knowledge has not been used in this context. We think this 
inclusion will lead to inductive learning in foreign language writing classes (Geluso, 2013; 
Acar, Geluso & Shiki, 2010, Yang, 2018). In this study, we have focused on the effect of 
teaching editing techniques in machine translation in an Iranian EFL academic writing 
context. Machine translation was specifically employed to improve the use of determiners, 
paraphrasing and collocations among our language learners. 


2. Literature Review 


Machine Translation (MT) has not found its way into foreign language classes as it should 
have. Bueno (1992) compares MT systems with handheld calculators and claims MT 
systems have not been fully integrated into language classes. If they were, we should 
need to think about how to use them and how to train teachers to use them as pedagogical 
devices. The majority of teachers forbid students from using such tools. However, it 
seems appropriate to teach students how to use these systems to improve their language 


learning. 
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So it seems to be appropriate to suggest an optimistic view regarding this technology in 
the classroom. Anderson (1995) describes a set of tools converting translation systems 
into powerful tools for language classes. Despite some studies into the use of MT in 
language teaching, (Garcia and Pena, 2011; Nino, 2008; Somers, Gaspari, Nino, 2006; 
Shei, 2002; Lewis, 1997; Anderson, 1995; Richmond, 1994), the findings are not 
conclusive and to some extent contradictory. For instance, Richmond (1994) focused on 
grammatical and linguistic drills with MT tools to increase students’ grammatical 
awareness. 


Lewis (1997) asserted that MT tools could be useful in language classes and both teachers 
and students should have access to them. Shei (2002) investigated the implementation 
of MT in learning a second language and teaching translation. He asserted that pre-editing 
was an effective technique that could enhance language teaching pedagogy. 


O'Neil (2012) also explored the effect of MT on L2 composition. He concluded that the 
quality of texts produced by students who used MT was superior to those who made no 
use of MT. Garcia and Pena (2011) conducted research on the use of MT and found that 
MT could be more beneficial to elementary learners than to intermediate and advanced 
ones. 


There are also studies claiming that MT could be detrimental to language teaching. 
Somers, Gaspari and Nino (2006), for example, admit that MT cannot be ignored due to 
its widespread use; however, they claim that assessment of student homework for 
language teachers is highly difficult, and students may project a wrong image of their 
true competence using MT tools. 


With the advance of online technologies, DDL has been accepted as a useful way to learn 
and teach foreign languages (Reppen, 2010; Granger, Hung & Petch-Tyson, 2002; 
Crosthwaite, 2017). The term DDL, as coined by Johns (1991), refers to a kind of 
language learning in which the learner is exposed to various examples in the target 
language, and tries to figure out what the correct form in the target language would be 
(Flowerdew, 2015). 


In DDL, the roles of both students and teachers are different from the traditional 
classroom settings. In this scenario, teachers are facilitators of learning, not the authority 
or expert, and students are guided through problem solving activities to raise their 
linguistic awareness and gain independence. So, when learners are exposed to numerous 
examples of a given linguistic topic, they will improve their self-editing skills and become 
autonomous foreign language writers (Todd, 2001; Reppen, 2010; Liu & Jiang, 2009). 


Some arguments have been leveled against DDL. For instance, it may not be suitable for 
all types of language learners. Flowerdew (2015) maintained that some learners may not 
benefit from this type of learning due to their previous learning background, learning 
styles, etc. For instance, studies have revealed that advanced and intermediate students 
seem to benefit more than elementary and beginner ones. Based on this finding, Liu and 
Jiang excluded lower level students from their study. It is interesting to note that few 
studies have used DLL with low-level students (Yoon & Jo, 2014; Chujo, Oghigian & 
Akasegawa, 2015). Likewise, some learning goals and learning styles have been 
considered to be detrimental to DDL (Yoon & Hirvela, 2004; Yoon, 2016). The question 
then is, if it has been proven that DDL is not beneficial for all language learners, would 
the use of ETMT help to open it up to a wider population of learners? 


To address such a gap in the literature, the present study addresses the following research 
questions using its mixed method research: 


1. What is the effect of teaching ETMT to Iranian foreign language writers in using 
determiners, paraphrasing and collocations? 

2. Compared to online bilingual dictionaries, how much use did the foreign language 
writers make use of ETMT for determiners, paraphrasing and collocations? 

3. How did the language learners use ETMT in their draft writing? 

4. What were the language learners’ attitudes and perceptions towards ETMT? 
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3. Method 
3.1. Participants and Settings 


The study was conducted in an intermediate academic writing course, Grammar and 
Writing I, in a university in Tehran over a five-week summer semester. The students met 
for three hours three times per week in a computer lab. The students were 20 Iranian 
foreign language writers who were enrolled in the course. They signed the consent form 
and participated in the study voluntarily. 


Table 1. lists the demographic profile of the 20 students (15 males, 5 female) from a pre- 
treatment questionnaire. The students’ age range was 20 to 26, with the majority being 
in their final year at the university. Their self-rated English score on the four language 
skills using a 5-point Likert scale indicated that they considered themselves poor in all 
language skills. Though the course was aimed at intermediate students, the proficiency 
test developed by TOEFL showed that the class included intermediate level students. They 
had no experience of living or studying in an English speaking country. In addition, they 
were familiar with Google Translate but not familiar with pre- and post- editing techniques. 


Table 1 


Demographic profile of participants in the study (N=20) 


Characteristic Mean SD Range 
Age 22.68 1.54 19-26 
TOEFL 580 1.00 310-677 
Writing 2.77 1.01 1-5 
Reading 3.21 0.87 
Speaking 1.88 0.91 
Listening 2.16 0.88 


Among various pre- and post-editing techniques in Machine Translation, we selected the 
most useful, namely, correcting mistakes, removing ambiguities, simplifying structures 
and combining structures. Based on the literature review, we expected the application of 
ETMT to be beneficial to these writers in their implementation of determiners, 
paraphrasing and collocations (Yoon, 2016; Shei, 2008; Park, 2010; Acar, Geluso and 
Shiki, 2011;). 


3.2. Procedure 


On day one, participants’ background information was collected using a pre-treatment 
questionnaire. An overview session on using Google Translate and editing techniques 
were offered to increase students’ awareness of how to use these techniques to improve 
their writings. They learned how to manipulate their input into the MT and how to edit the 
output generated by the MT. Subsequent days were arranged in this way: session 1 for 
determiners, session 2 for paraphrasing and session 3 for collocations. 


For each writing component, 30 minutes of training was offered with a pre-test conducted 
two days before and a post-test administered immediately after each session. In fact, 
Pre-tests 1 and 2 and post-tests 1 and 2 were divided into two five-item parts: Part A 
(error diagnosis and correction) and Part B (Persian to English translation). Pre-test 3 and 
post-test 3 contained five paraphrasing items. All the sentences and examples used were 
common mistakes made by Persian students. At the end, a post-treatment questionnaire 
was administered to obtain the students’ perceptions and experiences regarding the ETMT 
workshop. The students were asked to complete all the work online using Google Docs. 
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3.3. Data Analysis 


The data from this study was analyzed based on the performance on the tests and the 
responses to the questionnaires. As for the test data, the responses to pre- and post- 
tests 1 and 2 were labelled as O for incorrect answers and 1 for correct ones. Based on 
this coding, descriptive statistics as well as effect sizes were calculated. For pretest and 
post-test 3, a coding sheet was implemented to classify and quantify paraphrased 
sentences. 


Three categories were adopted for the evaluation, i.e., 0 for inappropriate vocabulary and 
grammar and 1 for appropriate vocabulary and grammar; 0 for different meaning in the 
paraphrased form and 1 for almost the same meaning in the paraphrased form and 2 for 
exact meaning in the paraphrased form; and finally, for the type of paraphrase employed, 
1 for different grammatical structure, 2 for different vocabulary form and 3 for both. 


As for the pre- and post-tests, the effect size of learning gains was calculated based on 
the benchmarks for foreign language writing using within-groups design (Plonsky and 
Oswald, 2014). The test data were coded and the inter-coder reliability was calculated 
using the Pearson correlation coefficient and reported as 0.89 


The remaining data were coded and analyzed using the following procedure: First, the 
students’ online bilingual dictionary pre-test and self-reported use of ETMT were analyzed 
to see how much they relied on ETMT. Second, the underlined expressions employed in 
the second drafts were classified into four types of changes compared to those they had 
used in their first drafts: new information, revision, correction and confirmation. These 
types of changes showed us how students used ETMT to confirm their original expressions, 
change wrong vocabulary and grammatical items, revise the expressions completely and 
add further information using new words and expressions. Frequency counts for these 
four types were also calculated for each student. 


At the end, student responses to the nine questions in the post-treatment questionnaire 
were split into three categories: computer knowledge and use (Questions 1-2), training 
in ETMT (Questions 3-6) and the use of ETMT (Questions 7-9). For the last question, the 
students were also asked to write about their experience in the course in English. 


4. Results 


Our data consisted of responses to the post-test questionnaire as well as test scores. 
Table 2 depicts descriptive statistics for both pre- and post-tests conducted in order to 
measure learning gain in the three training sessions. On average, in all three sessions, 
the post-test score was higher than the pre-test score. In particular, the use of 
determiners greatly improved (p<0.01) with the medium effect (d=1.14). However, the 
use of paraphrasing and collocations were less significant. The total gain score was 3.88, 
being statistically significant (t=3.85, p<0.01, d=0.88). 


Table 2 


Descriptive statistics for pre-tests and post-tests 


pre- post- 
test test 
Session | Mean SD Mean SD t p d 95% of 
Cl 
1 4.16 1.81 6.55 2.20 3.71 0.01 1.14 0.43- 
1.84 
2 6.62 1.87 7.12 7.13 1.13 0.27 0.28 -0-36- 
0.94 
3 11.05 3.18 12.16 12.16 1.16 0.08 0.44 -0.22- 
1.08 
Total 21.82 4.98 25.71 4.03 3.83 0.00 0.85 0.16- 
1.52 
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Figure 1 shows the percentage of using ETMT while completing the three post-tests for 
determiner, paraphrasing and collocations. Post-test 1 and 2 both included parts A and B 
(for error correction and Persian to English translation), while post-test 3 included only 
one part. Overall, the students reported that they used ETMT more than the online 
bilingual dictionary (72% versus 28%). They relied more on ETMT than the online 
dictionary to correct their erroneous expressions in part A of pre-tests 1 and 2. For post- 
test 3, they used ETMT more than the online dictionary but the difference was the 
minimum among the three post-tests (58.7% versus 42.3%). 


Figure 1 


he percentage of ETMT and online bilingual dictionary use across post-tests 


100 
= _ 


post-testi_A post-testl B post-test2_A post-test2_B post-test3 


ETMT wmBilingual Dictionary 


Table 3 presents the descriptive statistics for students’ answers to nine questions on the 
post-test questionnaire. Overall, the data revealed that the students’ computer skills and 
use were to some extent fair (M=3.84, SD = 0.66), and the ETMT training was effective 
(M=4.44, SD =0.64). As a result, they mentioned that they could and would use ETMT 
for their English writings (M=4.51, SD =0.65). 


Table 3 


Descriptive statistics for post-test questionnaire 


Question Mean SD 
Computer knowledge and 
use 
Before this course, | used 4.21 0.66 
computers a lot. 
| have good computer 3.43 0.71 
knowledge and skill 
Total 3.82 0.66 
ETMT 
It improved my writing in 4.62 0.51 
general 
It improved my correction 4.66 0.50 
Skill 
It helped me make sure 4.43 0.52 
my expressions were 
correct 
There was enough time to 4.05 1.01 
learn ETMT. 
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Total 4.44 0.62 
Implementation of ETMT 

| can correct mistakes 4.38 0.77 
| can remove ambiguities 4.38 0.60 
| can simplify structures 4.71 0.56 
Total 4.51 0.66 


5. Discussion 


Based on the questionnaire and test data, the four research questions in this study are 
discussed below. 


5.1. The overall effectiveness of teaching ETMT to improve writing 


The overall results indicated that the ETMT training resulted in a small-to-medium effect 
size (d=0.88) for teaching determiner use, paraphrasing and collocation. This effect was 
the highest regarding the use of determiners resulting in a medium effect size (d= 1.14). 
The discrepancy in learning gains among the three language components is an indication 
of how difficult it is to learn fundamental skills and knowledge to become good foreign 
language writers. It has to be noted that an incremental progress was considered while 
presenting these three writing components to Persian ELF students. As in many previous 
studies, our students were taught to use ETMT and were good at finding appropriate 
lexico-grammatical and word combinations in the given output (Park, 2010; Geluso, 
2013; Geiller, 2014; Conroy, 2010). They also learned paraphrasing by consulting MT. 
Learning gain in paraphrasing and collocation were lower than determiner use and this 
can be due to the fact that ETMT was useful in providing the correct determiner. Another 
explanation can be attributed to the fact that the majority of L2 writers (especially at 
beginning and intermediate levels) do not possess paraphrasing and collocation skills 
(Nesselhauf, 2005). For this reason, it can be concluded that an ETMT workshop should 
be of sufficient length to make a difference in the use of paraphrasing and collocation in 
foreign language writing (Eubanks, Yeh and Tseng, 2018). 


5.2. Perceived use of ETMT versus online bilingual dictionary 


The data for the use of ETMT and the online bilingual dictionary in pre-tests suggest that 
the students’ use of these two tools were different regarding writing tasks and 
components. While several studies have compared the effectiveness of using online 
resources, this study represented a real writing situation in which the Persian EFL learners 
looked up expressions and vocabulary items in online bilingual dictionaries (Yoon, 2016; 
Park, 2010). They were permitted to use online bilingual dictionaries during the ETMT 
workshop to resolve any problems during writing tasks. 


On the whole, the novice writers preferred ETMT to online bilingual dictionaries for error 
correction in both determiners and collocation. However, when working on more 
productive tasks such as translation from mother tongue into foreign language or 
paraphrasing, they relied more on online bilingual dictionaries. So it can be concluded 
here that ETMT can be more useful for controlled tasks than productive ones since it can 
provide the student with immediate corrective feedback. Students in controlled tasks are 
presented with sentences and phrases which are lexically and grammatically incorrect, 
and they are required to correct them using an online tool whereas in uncontrolled tasks, 
it is a challenge for the students to find problems and rectify them. In this regard, ETMT 
can be used as an error identifier if the errors are marked in the assignment (O’Sullivan 
& Chambers, 2006; Chambers and O’Sullivan, 2004). 


5.3. How ETMT was used for the revision process 
ETMT was used by the students in this study to revise first drafts of their writings. It was 
their first encounter with using ETMT and with how to use it for determiner use, which 


happens to be challenging for Persian students due to differences in the two languages 
(Persian and English). It turned out that many students used ETMT for confirming and 
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correcting their mistakes in addition to revising the whole sentence or enriching their 
vocabulary use. It has to be mentioned that students’ motivation, learning style and prior 
familiarity with this technology may also play a role. It can also be attributed to the 
novelty of the tool which makes it interesting for the students to use it. 


Another point worth mentioning is that intermediate level students used ETMT for revision 
and confirmation rather than correcting mistakes. This is in line with findings by Yoon 
(2016) whose advanced level students tended to use a variety of DDL-based materials to 
confirm their hypotheses instead of exploring new linguistic items. Based on this study 
and similar studies, it can be assumed that advanced learners with a good command of 
grammar and vocabulary employ these tools to confirm their hypotheses rather than 
using them to check their mistakes. This kind of approach to DDL self-editing techniques 
can encourage both inductive as well as deductive learning (Flowerdew, 2015). 


5.4. Perceptions and attitudes toward ETMT 


The post-treatment question data showed that all our students were happy with the ETMT 
training and also discovered that using ETMT instead of a search engine was more useful 
(Yoon, 2016; Park, 2010; Geluso,2013). Students proficient at using this technology 
reported that ETMT training was useful in their writings and they would use this tool in 
their future writings. Their views over the use of ETMT as a reference tool also showed 
that it increased their level of confidence in their writings. However, paraphrasing with 
ETMT was much more difficult compared to collocations and determiners since they had 
to provide alternative forms in the first language in order to be provided with acceptable 
paraphrases. 


Paraphrasing was challenging for novice learners since it included both collocations and 
determiner use. That is because these students had comparatively limited knowledge of 
structure and vocabulary in both languages. This can explain why some preferred using 
online bilingual dictionaries instead of MT. Thus, it must be mentioned that the 
effectiveness of ETMT mostly depends on the proficiency level of the students as well as 
the linguistic points in question which have already been mentioned by various 
researchers as critical factors in DDL (Yoon & Jo, 214; Franken, 2014). 


6. Conclusion 


This study was devoted to the effectiveness of ETMT in the use of determiners, 
paraphrasing and collocations among Persian L2 writers in an academic context. A 4-day 
ETMTworkshop was provided to the students with special focus on the use of determiners, 
paraphrasing and collocation. These areas were all challenging for the students who took 
part in this study. 


The qualitative as well as quantitative data revealed that two important pedagogical 
findings can be obtained from teaching ETMT in foreign language writing classes. First, 
detailed guidance by the teacher on the use of ETMT is crucial if students are to obtain 
maximum results from such online tools. Second, ETMT should be accepted as a fact by 
language teachers and incorporated into language teaching in general and foreign 
language writing in particular. The students should be taught how to edit and manipulate 
their input in order to attain best results. 


This calls for more research on the use of MT in various language learning settings 
particularly foreign language writing. Some avenues of research can be suggested to 
cover the limitations of the current study. First, longitudinal studies should be conducted 
to see how much training is needed and how long the effects of such training will last. It 
would be interesting to observe whether students use of these online tools would change 
and whether such use would be employed in other language skills and areas. Second, 
other factors such as learning style, task types and proficiency levels of students with 
regard to ETMT should be investigated. 


It seems that the key to successful implementation of DDL with regard to online 
technologies lies in how teachers introduce them to students. Almost all language 
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teachers are against MT and do not encourage its correct use. Discovery learning relies 
heavily on learner autonomy and to achieve it, training sessions should be conducted to 
turn learners into autonomous language users. Almost all DDL researchers agree that to 
make use of this technology, guided introduction is more successful than leaving the 
students on their own (Smart, 2014). Providing individualized guidance and feedback is 
crucial in achieving successful DDL learning. 
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